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ABSTRACT 

This paper discusses the rationale behind 
information-based approaches to school and systemic improvement and 
identifies some alternatives to the quality assurance model used, in 
New South Wales (Australia). It outlines the structures established 
for identifying and reporting issues and trends on a systemic basis 
and discusses some of the implications for the educational system in 
light of an analysis conducted in the first year of the quality 
assurance programs. The New South Wales Quality Assurance Model 
unites two functions of the school system — school development and 
accountability. The 1993 review report, which analyzed 70 individual 
school reviews, obtained data from interviews with 1,870 staff, 3,531 
students, 2,335 parents, 122 community members, and almost 300 
classroom observations. Findings indicate that, overall, schools set 
high standards of care for the communities they serve. However, the 
review suggests a need for development in several areas: the 
provision of system-wide resource allocation; improvement of 
participative decision making; improvement of student welfare 
programs; and development of a statewide assessment policy. (LMI) 
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Outcomes from Quality assurance School reviews: 
their use in system planning and decision-making 



Introduction 

Many of the attempts to improve the performance of schools and school 
systems over the past decade have been predicated on the assumption that 
improved planning and decision making will flow from the availability of more 
and better information about school processes and outcomes. This 
information-driven approach to performance improvement also underpins 
many of the current trends in business management, such as Total Quality 
Management. 

The Review Report 1993, which presents a synthesis of the findings of the 
Quality Assurance school reviews in the New South Wales (NSW) public 
school system, is a further example of an attempt to drive systemic 
improvement in education based on the provision of high-quality and 
comprehensive information about the performance of schools. 

This paper discusses the rationale behind information-based approaches 
to school and systemic improvement and identifies some alternatives to the 
quality assurance model that has been used elsewhere and in NSW in the 
past. The paper outlines the structures established for identifying and 
reporting issues and trends from the school reviews on a systemic basis and 
goes on to discuss some of the outcomes and implications for schools, 
regions and the system that resulted from the analysis of almost 300 reviews 
conducted in the first twelve months of the quality assurance program. 

REPORTING AND RATIONALITY AS A BASIS FOR SYSTEMIC 

IMPROVEMENT 

The belief that information changes things for the better underpins the work 
of most researchers in the social sciences. It is also a belief held dearly by 
many policy makers and adherents of rationalist decision-making models. 
They believe that information enhances the capacity to control, monitor, and 
evaluate, and, that as a consequence, improvements will follow from the 
ability to make better decisions and produce better outcomes. Information 



allows the individual to act rationally — to select the behaviour or set of 
circumstances most relevant to (his or her) presumed wants or needs 
(Bennett, 1964:84). 

This belief has perhaps gained greatest credence in the economic world, 
where it has been the driving force behind the push for availability of data on 
the national accounts and other economic indicators as a means of assessing 
current economic performance. The need for governments io have a clear 
understanding of the possible impact of policy initiatives in the light of 
changing economic and social conditions became apparent during the Great 
Depression of the 1930's, and has created the demand for accurate and 
timely data on such variables. A wide range of economic indicators are today 
regularly featured in news reports and the mass media, and are familiar to, if 
not understood by the community at large. 

In school education, the demand for public access to performance 
information has been slower to materialise. Until recently, increasing 
population growth and the drive towards universal literacy has largely 
protected the school sectors of developed nations from the need to monitor 
closely the impact of their activities. As a result, there has been relatively little 
information about the outcomes of schools and the ways in which these 
outcomes have been achieved. There are a number of explanations for the 
lack of information on school performance. These range from arguments 
concerning professionalism to the apparent difficulties of quantifying the 
outcome of education. These arguments have been buttressed by other 
dominant beliefs about th« nature and organisation of public school systems. 
In many countries, for example France, Australia, Sweden and the USA, it 
was assumed historically that the public schools were for all intents and 
purposes uniform, serving populations with similar needs and deriving the 
same benefits from common educational processes. These assumptions 
supported, and were supported by the concern to provide common inputs — 
best illustrated by extensive curriculum regulation, as in Sweden, monitoring 
of curriculum provision in Australia, the UK, France and The Netherlands, and 
the concern for fiscal equalisation in the USA (Ruby, 1991). 

In the context of the post-war expansion of education, the main data 
elements demanded by policy-makers were about inputs and processes. The 
use of tax revenues to establish and maintain public school systems also 
supported the demand for data on enrolments and completion rates — to 



justify the expenditure on grounds of fairness and efficiency. Policy-makers of 
these times had no need for quantitative information on school outcomes or 
processes of learning. Instead the focus was on quantifying resources, to 
ensure common inputs and measuring stocks and flows of students. There 
was no need to inform parents about the relative performance of schools or to 
provide for choice of schools because it was assumed they were essentially 
all the same. 

These organisational principles produced a managerial ethos for public 
school systems which demanded relatively little information on performance. 
Nor was there any need for a lot of information about processes because 
these were, largely, regulated to ensure uniformity. Monitoring of processes 
and performance was equated to apparent conformity with regulations and 
was carried out by political agents or inspectorates. 

These conditions persisted until the 1980's when a coalescence of forces 
saw a change in the dominant concerns of education policy-makers across 
the western world. Declining school-age populations, a general downturn in 
economic conditions and a shift in the prevailing ideological ethos have 
resulted in a greater demand for information about the outcomes of schooling 
— sometimes couched in the language of Value for money' of the services 
provided. Demands for comparative information about national school 
performance to address issues of international economic competitiveness and 
national efficiency have also contributed to this outcomes orientation. 

The changing conceptions of educational management provide a parallel 
explanation of the changing priority given to data collection and information in 
education. Levin, Windham and Bathroy (1990) argue that historically the 
dominant influence on educational management was "scientific 
management" . The first principle of scientific management, as enunciated by 
F.W Taylor in 1911, was that decision-making should not be intuitive but 
scientific. It should be based on observation, analysis and inference guided 
by the goals of efficiency. This principle continues to underpin many of the 
management models prevalent today. It is interesting to contrast the emphasis 
on statistical control in TQM models championed by Demings for example 
(see for example, Paine et a/, 1992) with that of measurement and 
observation in scientific management models. This is not to say that the 
implementation of TQM or any other information-driven model remotely 
resembles the oppressive and paternalistic implementation of Taylor's ideas 



as described by Callaghan (1963), but recognises that there are some 
common underpinnings to both. 

Levin et al argue that in the 1980s the dominant managerial model has 
been "applied science" where available information is used to identify and 
define problems and then select a course of action. This model which 
subsumes the "reflective practitioner" uses scientific information as well as a 
common sense, and practical knowledge. They believe that this model links 
information and the quality of education and extends the "demand for the 
development of more extensive and more imaginative educational information 
systems than planners have used in the past". (1990: p78) 

A number of major reform programs that aim to provide better information 
about schools to improve management, accountability and parental choice 
have been initiated in many countries. Ruby (1991) argues that these deserve 
closer examination because they- represent a significant and substantial break 
in the prevailing doctrine about the information and data in education. They 
also bear directly on ideas of accountability and illuminate wider policy 
questions, such as who controls information about the functioning of schools 
and where the responsibility for poor performance is located. 

When we speak of information-based models of decision making, we are 
also speaking to a large extent about evaluation. Information is more than raw 
data, but is data to which a particular meaning or value is inferred. The 
method of reporting is only one half of the equation in this approach to 
systemic improvement — the means of generating the information is equally 
as important. Often the two go hand in hand, with the form of reporting being 
dictated by the kind of information generated by the evaluation methodology 
employed. A number of evaluative models have been used at various times 
and in various contexts in attempting to drive systemic improvement in 
education systems. These models fall into a number of categories, which 
include (1) accountability-based approaches; (3) self-studies; (4) the 
effective schools paradigm; (5) school recognition or lighthouse programs; 
and (6) models based on rich, contextualised descriptions of schools. These 
models differ in important ways and the extent to which they have proven to 
be of enduring value, Haertel, Katzenmeyer and Haertel (1989) in reviewing 
the rationale, purpose, and inspection and accreditation systems; (2) 
performance indicator systems; (3) strengths and weaknesses of several of 
these approaches argue the following as essential conditions for the 



and weaknesses of several of these approaches argue the following as 
essential conditions for the comprehensive evaluation of schools. The 
same conditions hold for the evaluation of school systems. 

"In order to diagnose problems in a school's functioning and at the same time 
suggest possible avenues and approaches for improvement, a comprehensive 
evaluation must examine the school's instructional processes, as well as its formal 
and informal administrative and decision-making processes. It must elicit the 
views of the school's participants concerning problems and solutions, as well as 
inferring problems from evidence of poor learning outcomes, negative parent or 
community sentiment or other sources. Above all, these different forms of 
information and evidence must be brought together to create a coherent picture, 
a sort of casual model of the school's functioning. Mere description will not 
suffice" (p2) 

in the light of these, conditions, the following section describes the NSW 
Quality Assurance model and provides support for this present approach to 
improved decison making. This process is contrasted with two othrer models 
for providing policy information — those provided by the establishment of 
Inspectorates and those based on performance indicators. 

The NSW Quality assurance model 

In 1992 the NSW Government announced its commitment to a program of 
quality assurance reviews of schools and the services and programs provided 
to schools. Quality Assurance is a new function in the NSW Department of 
School Education. While there have been many means by which schools and 
education systems have attempted to demonstrate accountability in the past, 
quality assurance is based on a different approach that unites two distinct 
functions of the school system — school development and accountability 
(Cuttance, 1993a and 1993b). 

This approach to quality assurance addresses the charge provided by the 
Director-General, Dr Ken Boston, that we make sure, be sure, and give 
assurance to others about the quality of our work in education. Quality 
Assurance is — or should be — an integral part of all our work throughout the 
school system — it is about building— in quality. That is, continuous 
improvement must become part and parcel of everything that we do. This 
approach to quality assurance differs in both philosophy and practice to 
previous models of quality control in education which are based on 



inspecting-out inferior outcomes after they have been produced. Quality 
assurance seeks to prevent performance problems arising in the first place. It 
is therefore one of the cornerstones of a quality system, a system for 
managing the quality of what we are attempting to produce. This model of 
quality assurance adopted in NSW differs from that used in industry and 
indeed other education systems in both philosophy and implementation, 
although it does draw considerably on the South Australian School Review 
Model described by Cuttance (1990). 

Quality assurance school reviews are only one part of the entire quality 
assurance strategy. School reviews assess the effectiveness of practices and 
processes for achieving improved student outcomes in schools. They 
contribute to the process through which schools as learning organisations 
develop a planned approach to constructing their future. 

In NSW the quality assurance school reviews are a collaborative 
enterprise, in which review teams work with school communities to assess: 

• how the school goes about the task of meeting the community's needs 
for education, in the context also of addressing statewide priorities for 
student outcomes; 

• what the school is attempting to achieve for its community; 

• how the school knows it is achieving what it set out to do; 

• how the school responds to what it knows about its achievements. 

The need to assure quality does not mean that there is something 
inherently wrong or deficient in the current operation of schools. Rather it 
reflects the need to provide a clear framework for public accountability and 
provide the information about current performance required to improve 
performance further. 

Quality assurance reviews of individual schools focus on the way in which 
the school is providing for its community's education needs. They are 
concerned directly with student outcomes and the quality systems in place to 
bring about their improvement, through effective management and leadership 
in schools, communication and community participation within the 
perspectives and constraints of statewide priorities and resources. Integral to 
the concept of quality assurance is the community's ownership of the review 
and development process. In the review process, staff, students and members 
of the school community participate with a review team to assess the current 
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achievements of the school and the types of activities that would, through the 
planning and development process, contribute most to future achievements. 

The starting point for all school reviews is the development plan for the 
individual school. This plan, developed collaboratively by the school 
community to meet local educational needs, forms the basic terms of 
reference for the review process. The primary focus areas for the review are 
negotiated with the school community. 

School reviews also serve an accountability function by providing the 
community and the Department of School Education with public information 
about the success of programs and initiatives of schools. Further, they place 
on record the key issues that individual schools will give priority to in revising 
their plans for future development. 

School reviews are conducted by teams of size and composition which 
vary according to the enrolment of the school. The team leader is responsible " 
for setting up the review, managing it during the period in the school and 
producing the report of the findings and recommendations. The host school 
principal is involved throughout the review, and participates as a full member 
of the team. For public accountability purposes it is important that the review 
team contain members from outside of the school. In general, review team 
members are peer principals, executives or teachers from other schools. In 
addition, the review teams contain at least one parent or member of the local 
community. 

Each review results in a written report to the school within 20 days of 
completion of the review. The reports conclude with a number of 
recommendations for the strategic development of the school. The individual 
school review reports provide an extremely rich source of policy-relevant 
information for the education system as a whole. When aggregated on a 
state-wide basis, the findings and recommendations of the reviews reveal 
trends and issues emerging in the management and operation of schools 
which have the potential to significantly influence policy formulation and 
resource allocation. 
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Other models of quality control 

Quality assurance as practiced in NSW is only a relatively recent 
phenomenon. Historically, the most important means by which education 
systems obtained information about themselves has been through the 
establishment of corps of Inspectors, who have reported on a school-by- 
school basis to the Director of Education. NSW has a rich tradition in the 
quality control of schools through school inspectors, beginning even before 
the appointment of William Wilkins as the first Inspector and Superintendent of 
Model Schools in 1854, until the abolition of the inspectorate in 1990. Until 
recently, most other Australian states also had similar inspectorial services. 

From the very early days, formality was the keynote of the NSW education 
system. Initially, the chief role of the inspector was as an examiner. 

With poorly trained teachers to contend with, his method was to whip his 
teachers along by severity and criticism until they achieved the maximum 
possible percentage of passes in the Standards of Proficiency (Logan, 1984: 
p2). 

While school development was seen as a part of the Inspector's role since 
at least 1904, it is fair to say that inspectors never really became the friend, 
guide and adviser that one of the early Directors of Education, Peter Board, 
envisaged them to be. Rather, assessment of teacher's proficiency continued 
to dominate the workload of inspectors for most of their existance. 

For a brief period in the early 1 970's, whole school appraisals entered into 
the Inspector's duties. These were to be a formal appraisal of a school by an 
inspector or inspectors, which took into account the evaluation made by the 
principal and staff, and were to be followed by a written report for the 
information of the Director-General at intervals of no more than six years. The 
inspector was to have detailed information about successes and failures in 
setting goals, selecting methods and constructing tests which he [and 
inspectors were almost always male] could discuss with the teachers 
concerned so that he could form his own ideas about their quality. 

His attention will be riveted on process and outcomes not on personality. 
The attempt here is to involve schools and teachers in their own appraisal and 
so rid the system of the fear and mistrust which is generated by an imposed 
method (Grassie 1971 in Logan, 1984). 
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A number of appraisals were carried out. Some were reported to be quite 
successful, but by 1979, Logan notes, appraisals were as good as dead. We 
need not trace here the reasons for the fall from grace of either the 
inspectorate as a whole nor the school appraisal schemes which showed 
such promise, except to draw attention to the complex political and industrial 
circumstances prevailing at the time. What is important to note is that even if 
the school appraisals had been successful themselves, no formal 
mechanisms were established that could have made it possible for system 
wide improvement to follow. School appraisal, on its own, was doomed to 
failure as an agent of large scale change. 

David Hargreaves (1987), writing in the context of Chief Inspector of the 
Inner London Education Authority (ILEA) in the late 1980's, commented that 
he considered the two functions of monitoring the quality of education and 
improving the quality of education to go hand in hand. As a result, he 
identified three central problems in carrying out this work as: 

1) what information is needed centrally by the authority; 

2) what is the purpose of collecting the information; and 

3) to whom is the information to be reported. 

He went on to outline the history of performance monitoring in ILEA. While 
written in that specific context, the weaknesses he highlighted are generic to 
most inspection systems. 

We had the tradition going back for most of the century of full inspections, which 
gave a very detailed report of the work of the school. When the HMI decided to 
publish the results of their inspections, we decided to publish the results of our 
inspections too. These reports were made available to the press, parents and the 
public. If it was a favourable inspection report, full of praise, you would find that 
principals had them at the entrance to the school, available for any casual visitor. If 
the report was not so favourable they are usually kept in a cupboard in the back of 
the school. 

There are weaknesses in this approach to monitoring. This model is satisfactory 
for the HMI because they are drawing on a sample nationally, and although the 
sample is very small it is probably large enough for them to be able to make 
general statements about overall conditions. 

In the ILEA I did not have enough information to make such statements each year 
even though I increased the number of inspections. The chances of a school 
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being inspected in London under the full inspection model are about 1 in 50 in 
each year, hardly enough to strike terror or even apprehension in any principal. It 
is a terribly weak base from which to generalise. How could I draw on a sample of 1 
in 50 and make general statements about the quality of education in the system? 
A second disadvantage was that this model is not as helpful for developmental 
purposes as we would have wished. It took about a year to get the full school 
inspection report printed and made public, by which time conditions in the school 
had most likely changed. Of course we gave the schools oral feedback as soon 
as the inspection was completed but the gap of a year to complete the written 
documents was too great. There is a third problem, and a very important one. This 
system of sampling does not identify your weakest schools, the schools in 
difficulty, the schools that are being poorly led, or the schools where there is 
serious under achievement. If you are taking monitoring seriously, samples will 
not do. You need to identify every school that needs help. This kind of approach 
using full inspections, though it has some value was not cost effective.". . 
(Hargreaves, 1988 p.127). 

In a third model of school improvement — school self-evaluation — ILEA 
was also considered one of the pioneers in the field. Schools were guided by 
the document Keeping the School Under Review in order to make 
judgements about themselves with some assistance from inspectors. This 
proved not to be a terribly effective system either. Nor were similar systems in 
other areas any more effective. Hargreaves found that schools tended to write 
long accounts of what they had been doing which were heavily descriptive 
and heavily defensive. Teachers were not good at analysing what they were 
doing. These school self-examination reviews were extremely weak at 
examining what went on in classrooms. They were very weak at analysing 
pupil performance. The final weakness was that schools were not very good 
at forward planning, which was often couched in the vague and conventional 
rhetoric of education of the time. 

Hargreaves concluded from this experience that there are four features vital 
to any central monitoring system, whether established as an inspectorate or 
not. First it has to be driven by credible people. They must be credible not only 
to people in schools but also in the community. Secondly, they must be 
trustworthy. They will receive a lot of confidential material, some of which itself 
is not made public, although it may be used to contribute to some broader 
picture that will be. These first two features suggest that the people involved 
need a degree of bureaucratic independence from the system. The third 
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feature is that they must combine the two functions of monitoring and 
development. Finally, they must be people who can really act to encourage 
and support the work that principals and teachers do. They must be people 
who can give praise and give help. Simply having some indicators or greater 
accountability will not automatically improve schools. These ideas have 
become key principles in the operation of Quality Assurance in NSW (see 
Cuttance 1993b). 

The ILEA experience had one other important legacy for shaping new 
ideas about ensuring systemic quality. In a short space of time the system of 
school inspections (called substantial visits) had generated thousands of 
reports from the system of school visits established as the means of 
overcoming the weaknesses of self-appraisal and the old full inspections. 
This system was great on reporting back to the schools themselves, but it was 
weak on coining central information. An annual report was produced for the 
politicians to give an account of the substantial visits, and to begin to quantify 
some of the judgements about what had been seen. 

To do this a rating scheme was established to quantify aspects of school 
life on a four point scale, ranging from outstanding to satisfactory, not 
satisfactory, to very poor. There was no middle category. Inspectors were 
asked to make a very hard judgement about what was satisfactory, and what 
was not. Each lesson observed was rated, as was the effectiveness of 
departments in secondary schools, the management and organisation of the 
department by its head, the quality of the accommodation and resources, the 
quality of schemes of work, the capacity of the schools to differentiate to meet 
the needs of pupils who were very able and those with special education 
needs, and so on. These were quantified in an annual report which gave an 
overall picture.. 

This report was a very substantial document, which for the first time for the 
ILEA gave a picture of the whole system, and gave something for principals to 
compare their school in relation to the average. Hargreave's systemic report 
then, is primarily a presentation of quantitative information, even if based on 
qualitative judgements. 

The mode! of school review and reporting closest to that adopted by Quality 
Assurance in NSW is the review of the total provision of Local Education 
Authorities (LEA) by Her Majesties Inspectorate (HMI) in the United Kingdom 
during the mid 198G's. These reviews commented on a wide variety of areas 
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of operation of the LEAs, including consideration of policies, consultation, 
budgeting, ethos, standards, curriculum offering and organisational patterns. 
The strength of the HMI reviews lies in their ability to synthesise information 
from schools across the LEA and to link school-derived information with 
commentary on the support structures in place at LEA level. They also 
demonstrate that it is possible to sensitively and constructively report on a 
system wide basis. 

There are also major differences between the Quality Assurance and HMI 
reviews. Not least is the difference in scale of school population and 
geographic size between NSW and Local Education Authorities in the UK. 
The major difference is the internal frame of reference which drives the 
review. The HMI reviews are much more structured and less negotiated than 
the Quality Assurance school reviews. They also differ in the regularity and 
frequency of the reviews — it is possible that an LEA would not experience 
another full review within a decade, if at all. They also differ in composition of 
the review team (HMI being completely external to the LEA), and thus have 
limited potential to offer the developmental experience to school-based staff 
which underpins Quality Assurance and in the extent to which they are able to 
provide guidance for individual schools within the review process. 

ILEA, of course, is no more, and even the role of Her Majesties Inspectors 
has now been radically redefined. Other approaches to system wide quality 
control have come and gone. Performance indicators have been embraced 
with much enthusiasm. The following section examines this concept briefly. 
While it will be seen that they offer some advantages over inspection based 
systems, there are also distinct shortcomings in their usage, which will be 
examined below. 
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The performance indicator approach 

Performance indicators provide an approach to information management 
and reporting which has captured a great deal of attention in education 
systems in the last decade (see Wyatt, 1991 and 1992 for a comprehensive 
discussion of this movement). By the late 1980's almost all states within the 
US have implemented indicator-based accountability systems of one kind or 
another (Selden, 1987). Indicators have become adopted as part of the 
national education reporting strategy in countries as diverse as Portugal 
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(Climaco, 1992) the USA (Griffith, 1990) the Nordic Group, and by the OECD 
itself (Bottani and Delfau, 1990). 

Indicator schemes report on a wide variety of variables, and usually include 
measures of inputs, such as pupil-teacher ratios and per-pupil expenditures, 
processes and outcomes. Variables representing educational outcomes are 
prominent in all indicator systems, and these most frequently feature student 
test scores in one form or another. Because educational outcomes are 
strongly influenced by some factors beyond a school's control, most also 
report background data of some kind to help guide reporting and use of 
outcome variables. 

Indicator systems have several features that could allow them to contribute 
positively to comprehensive school improvement efforts, and indeed have 
been a driving force in many examples of systemic reform (eg Fetler, 1986). 
They can provide benchmarks for comparing educational progress over time 
and place. They can help to describe changes in key features of the education 
system and direct attention to present or potential problems and inform policy 
decisions (Oakes, 1986). 

However, there are several difficulties with indicator systems as models for 
comprehensive evaluation. They employ a limited range of variables, and 
tend to focus excessively on outcome measures, especially objective test 
scores, rather than measures of education processes. Despite their stated 
intentions, most do not provide any coherent model of a school's functioning, 
and so are of limited value for either describing schools or diagnosing their 
problems. Finally, although some of the more successful accountability 
systems have been designed and operated at the local school level (see 
Wyatt, 1992), those managed at the state level are unlikely to promote the 
kind of systematic, cooperative effort among teachers and principals that is 
needed to effect significant school change because they are often perceived 
as imposing irrelevant, additional demands on schools (David, 1987). 

Most indicator systems rely primarily on data already obtained for other 
purposes. This is both a strength, in that the cost of information gathering is 
reduced, and at the same time also a weakness. Because these various data 
collection activities have been initiated for different purposes and 
administrated under different auspices, there is potential for poor coordination 
of data collection or integration of the information collected. In the United 
States, for example, the development of educational accountability systems 
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has led to some consolidation of state education data, but these data have not 
been adequate to create systematic, coherent, and comprehensive indicator 
systems. The variables included may be informative in themselves, but taken 
together they often fail to address important aspects of school structure, 
function, and outcomes (OERI State Accountability Study Group, 1988). 

From their study of indicator systems in 50 states, Kaagan and Coley 
(1989) conclude that indicator results are being used prematurely to hold 
local schools accountable; that there is insufficient investment to insure high 
quality of the measurement that becomes part of the state indicator system; 
and that there is a reluctance to stress understanding of the indicators, that is, 
to postulate relationships between inputs and outcomes for the purpose of 
recommending policy action. 

Odden (1990) underlined the need to be clear about the purpose of an 
education indicator system and not to use it inappropriately. He saw the 
biggest weakness in indicator practice as the need for better interpretation of 
what the indicators portray. Both Kaagan and Coley (1989) and McDonnell 
(1989) argue that using indicators for accountability is still premature because 
few systems yet provide enough information and there is still too much to 
learn about the linkages of cause and effect between indicators. Current 
systems also provide data too far removed from the classroom level to 
improve local conditions. These authors conclude that the indicators are 
therefore best used to offer policy makers a broad overview of the education 
system. 

None of these reasons are sufficient in themselves to suggest that 
performance indicators have no place to play in the strategic management of 
education systems. However, it is also obvious that neither the indicators or 
the uses to which they are currently put provide decision makers with the 
information needed. The best solution, as Porter (1993) suggests, is likely to 
be a mix of indicators and context sensitive information about school 
processes. 

The Quality Assurance plan for systemic reporting 

To complement the individual reports produced as a result of the school 
review process, the Quality Assurance Directorate intends to institutionalise a 
process for bringing synthesised information from the Collected reports into 
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the policy-making arena for the education system as a whole. The framework 
for this systemic reporting contains several important elements. 

First, the reporting will be tailored for several levels of decision-making 
across the system. Reports synthesising information from the school reports 
will be prepared for regional authorities (the Assistant Directors-General 
(Region)), State Office directors, and the Director-General. (The Quality 
Assurance Directorate also reports to the Minister and the public through the 
Department's Annual Report). Secondly, the framework recognises the 
importance of timeliness for the delivery of information. The time frame for 
reporting to each of these levels differs, with reporting to those closest to 
schools occurring most frequently. Reporting to regions and state office 
directors every six months and to the Director-General every 12 months. 
Third, the framework recognises the different needs for information at each 
level. Information for regions will be drawn largely from the recommendations 
for development made in each school. Greater emphasis on implications 
drawn from the review findings and school context statements will be more 
appropriate for the needs of state office personnel. The main vehicle for 
information transmission is through formal written reports, although the reports 
to the Regions and state office directorates will be delivered in conjunction 
with a verbal briefing and informal discussions held with senior staff members. 
These reports to regions and state office directors will be public reports. 

While each of these reporting strategies are important, the remainder of this 
paper describes in greater detail the method of analysis, the findings and the 
outcomes of the publication of the Directorates major public reporting vehicle 
in its system-wide Review Report. 

The Review Report 1993 

The Review Report 1993 is in many respects a unique document and 
represents a significant milestone in the development of the Quality 
Assurance Directorate as a vital and integral part of the education 
bureaucracy. The first edition, while in many respects a pilot venture, was 
important in demonstrating that it is possible to report in an accurate, 
meaningful and comprehensive manner on the status of schooling across the 
education system. The document is unique in that this is the first time that the 
Department has published a detailed account of both the processes and 
outcomes of schooling. Together with an analysis of areas of operation 
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requiring further development, this information provides a powerful indication 
of the strategic directions for improvement across the system. It is also unique 
in that while Directors-General in the past have reported to parliament 
through Annual Reports, these tended to favour description of input to and 
resource deployment across the system rather than an analysis of its 
performance. 

The first of the systemic reports is based on the trends and issues arising 
from the findings and recommendations of the school reviews conducted in 
Terms 1 and 2, 1993. School reviews, as described earlier, form the 
cornerstone of the Quality Assurance process, but are by no means the only 
part of the Quality Assurance Directorate's review activities. Future editions of 
the review report will also incorporate summary information and commentary 
on the findings of program evaluations and reviews of Best Practice 
conducted by the Directorate. These activities are described in Highett, 
(1994). 

As the compilation of the Review Report is such a significant undertaking, 
both in terms of its potential impact and in the magnitude of the undertaking, it 
is worth spending a little time outlining the processes involved in its 
construction. 

While qualitative methods have been an accepted tool in educational 
research since the late 1970's (eg. Guba 1978), practical guidance on the 
mechanics of qualitative data analysis (as opposed to data collection) is 
comparatively thin in the literature. Firestone, and Dawson describe the 
under-development of data analysis techniques as perhaps the major 
stumbling block to further use of qualitative methodology. Miles (1979) cites a 
study by Seiber which found that only 10 percent of the content of the better 
textbooks on qualitative methods was devoted to issues of analysis. 

The potential for "fuller appreciation of the phenomena of interest" (Eisner, 
1979) offered by qualitative methods must be balanced by the need for 
studies to be conducted in a disciplined manner which can withstand external 
scrutiny. Any information offered to decision-makers which proposes 
directions for action requiring significant resource shifts must have a high 
level of credibility if it is to have any chance of acceptance. In constructing the 
Review Report, it was essential, therefore, that the data analysis was able to 
demonstrate a high degree of methodological rigour. 
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Firestone distinguishes three general approaches that may help discipline 
qualitative inquiry without sacrificing subjective understanding — intuitive, 
procedural and intersubjective approaches. Each of these approaches has 
strengths and weaknesses and contribute differently to the research process, 
but in practice the three approaches are generally used in combination. 

The intuition of the individual analyst is sometimes described as the richest 
and primary source of subjective understanding in qualitative research, as the 
researcher immerses themselves in the rich and thick data resulting from the 
transcription of field notes, interviews and records of observations. The 
weakness of intuitive approaches is that because intuition is such a private 
process, it is difficult to convey the results to external scrutiny or confirmation. 

To provide the credibility public reports require, we are therefore forced to 
adopt a more procedural approach such as that offered by content analysis 
(see for example Berg, 1989) in which a hierarchy of significance is 
constructed, based on the frequency with which an issue occurs in the data. 
Content analysis itself is sometimes criticised as approaching the positivistic 
(the antithesis of the naturalistic philosophy), as it requires pre-determining 
the concerns or hypotheses that are to be tested against the data. 
Alternatively, grounded theory (Glasser and Strauss, 1967) provides a 
strategy for generating hypotheses from systematically acquired data. One 
element of the grounded theory strategy — constant comparative analysis — 
involves ongoing, systematic organisation and classification of the data into 
various categories. When categories of data or themes begin to emerge, other 
data that fit the category and other related sub-themes can then be discerned. 

For the first edition of the Review Report, the organising framework chosen 
for the analysis draws on both the content analysis and grounded theory 
approaches. The focus areas for review were selected as the primary unit of 
analysis, and the four most frequently occurring areas used as the major 
headings under which information would be reported. These areas were: 
School and Community; Student Welfare; Curriculum, Teaching and Learning 
and School Planning and Decision Making. 

Within each of these areas, we initially identified practices that were the 
subject of particular commendation, findings in relation to strengths or areas 
in need of further development, and directions for changes suggested by the 
recommendations. Within each focus area, themes in these findings were 
identified and tallied, and linkages between themes sought. Illustrative 
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examples under each of these headings were noted and used to further 
illuminate the issues identified. To keep the report within manageable 
bounds, only those issues that were noted in a considerable number of 
schools (in this instance at least one-fifth of the schools in the sample) were 
included in the final write-up. From these issues, implications for various 
levels of the system were then identified and directions for further 
development suggested. 

Alternative analytical models 

The analytical framework adopted for this report is not the only one 
possible. One of the problems encountered in imposing a structure on the 
analysis based on the titles of the focus areas is that often these titles cover 
different content in individual reviews. For example, a review of "Decision 
Making" in one school may have covered very similar ground to that in 
another school which might have been labelled "Management Structures". 
Likewise a review of "School and Community Relations" might have much 
more in common with a review of "Communication Processes" in another 
school than similarly titled reviews elsewhere. 

Other schemes for categorisation and therefore other units of analysis are 
possible and will be explored in future syntheses. One method is to adhere 
more closely to the principles of the grounded theory approach and thus 
identify themes or topics as they emerge from the analysis as it progresses 
without any prior categorisation. Alternatively, a different framework for 
content analysis might prove more appropriate, for example by developing a 
priori a set of topics of interest to various audiences and then searching for 
information about each of these from the data. A third possible analysis would 
use the Best Practice Statement dimensions, areas and aspects as the 
categorisation framework (see Highett et al, 1994). 

Our intention is that the work we undertake and the documents we produce 
will not only be considered useful by our client audiences but also used by 
them. In the spirit of kaizen, we seek continual improvement of our own 
processes. The direction of future analyses will thus be based on the 
feedback from our clients as well as further consideration of emergent 
methodologies. 
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Use of technology 

One of the greatest disadvantages of any qualitative data analysis is that it 
is of necessity extremely labour intensive and relies on the subjective 
judgement and expertise of the analyst. The compilation of the Review Report 
1993 was no exception. Each of the individual school review reports typically 
ran to about 30 pages in length. Even a cursory scan of each copy took 
several minutes to complete, and one reading was scarcely sufficient to 
unearth all of the information pertinent to any topic or to understand the 
nuances of context and relationships between issues which were embedded 
in the reports. When fully operational, the systemic reporting process will draw 
on around 15,000 pages of data for each reporting cycle. 

Clearly, even though systemic reporting is the responsibility of the 
equivalent of one full-time officer, it remains a Herculean task in the time 
frame required when conducted as a manual exercise. Unlike quantitative 
data analysis, where the ready availability of desktop computer processing 
capacity has made possible the almost instantaneous analysis of huge data 
sets, there is no equivalent software that substantially reduces the burden of 
qualitative analysis. While there are at least ten specially written computer 
programs designed to handle the storage of coded data for analytic purposes, 
none has yet proven entirely satisfactory for our purposes, primarily because 
the necessity for coding of data is almost as time consuming as manual 
methods. As yet, computers do not analyse or interpret qualitative data. We 
concur with Ely (1991), who noted that although the software packages can 
remove most of the drudgery from the cut and paste process, the computers 
are unable to do the actual analysis for the researcher. However, Richards 
(1990) has designed a package called NUDIST, which not only codes and 
retrieves text, but also "assists the users in shaping their understanding of the 
data, and helps them form and test their theories about the data". We will 
watch the further development of this and other systems with interest. 

Some commentators on qualitative methodology (Pfaffenberger, 1988; 
Fielding and Lane, 1981; Merriam, 1988) take a very cautious, if not entirely 
negative view of the use of technology in analysis. They believe that 
researchers and their audiences may become seduced by the convenience 
presumed credibility of the program's rendering of sense, without 
understanding the assumptions implicit which, while present in all forms of 
qualitative analysis, are often hidden from view in computer software. 
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Notwithstanding these dangers, in practice we are faced with a significant 
data management problem, for which a technological solution has several 
long-term advantages. For the next cycle of analysis we will be trailing the 
use of a commercially available software package called IDAS (Intelligent 
Document Analysis System). IDAS has several features which will relieve 
some of the more mundane tasks in analysis. Firstly, it allows the analyst to 
create summaries of documents at a level of summarisation set by the 
researcher. Second, it creates an index of the most frequently occurring 
themes within a document and allows the user to transfer from this index to 
the places in the document where the desired word, theme or concept occurs. 
It is possible for the user to customise this word search to a certain extent. For 
example, similar information might be indexed under several terms, such as 
assessment, testing and examination and so on. We are not yet in a position 
to be able to evaluate the effectiveness of IDAS for our analytical purposes. 

What the first report found 

The first systemic Review Report is based on the findings and 
recommendations contained in 70 of the 93 individual school reviews 
completed at the end of Term 2, 1993. These school reports collectively draw 
on interviews with 1870 staff, 3531 students, 2335 parents, 122 other 
community members and almost 300 classroom observations. 

The review reports published in Terms 1 and 2 documented many 
examples of sound practice, but also revealed that there was great variability 
in the effectiveness of practices both within and between schools. Several 
areas of commendation were noted. Strongest among these was the very 
positive perception in the community of teachers, principals and programs in 
the majority of schools reviewed. The efforts of schools to ensure the welfare 
of students in their care was also noted for commendation in several 
instances. Many schools reviewed thus far could justifiably take pride in the 
academic, sporting and other extra-curricular achievements of their students. 

Student welfare was a focus for review in many schools. The review teams 
noted a very large number of programs instituted by schools to address the 
needs of their diverse student bodies. On the whole, these programs were 
perceived positively by school communities. However, an issue emerging in 
many schools was the perceived inconsistency of application of discipline and 
merit award schemes. Also apparent was a degree of concern about the 
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effectiveness with which information about these programs is communicated 
to parents. Another frequently occurring theme was the need for greater 
student involvement in decision making in many schools. 

Reviews of curriculum and teaching and learning took two forms. In primary 
schools the focus was most often on the implementation of particular Key 
Learning Area syllabuses (with English, Mathematics and Science and 
Technology the most frequently nominated). In secondary schools the focus 
was more frequently on the structure and organisation of the entire curriculum 
offering. The main areas requiring further development were the identification 
of expected student outcomes, the coordination of sound practice in 
evaluation and assessment, particularly the use of outcomes to drive forward 
planning, and the adequacy with which assessments of student progress is 
reported to parents. There was an awareness in schools of the need to match 
student learning styles with appropriate teaching strategies. In response to 
growing student retention rates, almost all of the secondary schools reviewed 
had begun a self-examination of the content and organisation of their 
curriculum offering, especially at the senior secondary level. 

In the area of school and community relationships, it was clear that while 
there was considerable parent involvement in schools in terms of working in 
the canteen and classrooms, for example, and there was evidence of 
increasing participation in management issues, there is as yet relatively little 
parent participation in strategic or long term planning. School Councils, where 
they exist, were often still at the stage of refining their roles and 
responsibilities. Recommendations in this area frequently expressed 
encouragement for them to seek a deeper understanding of the concept of 
participation, to seek wider community participation and to enhance 
communication with the entire school community. 

In school planning and decision making, the effectiveness of practice 
across the system emerged as a matter requiring further attention. Only a few 
schools were commended for exemplary practices in planning. Some schools 
did not yet have adequately documented plans and others did not use them 
as working documents. There was a need to strengthen the strategic focus of 
school plans identified. There also needed to be more substantial 
involvement of all stakeholders in the development of plans. The extent to 
which the planning process is information-driven and clearly linked to desired 
student outcomes was another area requiring further attention. 
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Other issues raised were often matters primarily of importance to the 
particular school, such as particular resource or properties matters. A 
mechanism for bringing these issues to the attention of the appropriate 
authorities has been established to complement the school review reporting 
process. Issues raised in this manner which have wider significance were 
very similar to those made in other schools where the subject was a focus for 
review. 

What were we able to say to the system 

The Review Report 1993 has been able to satisfy at least part of the 
requirements of quality assurance that we make sure, be sure and assure 
others of the status of public schooling in NSW, in that it quite clearly provides 
a very public accounting of the strengths and weaknesses of the sample of 
schools reviewed during the review period. Although this first effort was based 
on a fairly limited number of self-nominating schools, and this shortcoming is 
recognised in the analysis, future editions will each draw on over 500 
individual reviews of a cross-section of school, or one quarter of the schools 
in the NSW public school system each year and thus provide very substantial 
evidence of systemic performance. 

The Review Report has been able to assure others by being able to 
substantiate the many fine achievements of schools across NSW. Schools 
reviewed were seen in the main to be setting high standards of care for the 
communities they serve. There was evidence that schools are moving towards 
becoming an integral part of the wider community. 

These achievements notwithstanding, the synthesis of findings from the 
school reviews indicated the need for further development in several areas, 
which had implications for different levels of the education system. While the 
recommendations made in the review reports relate to the individual school 
communities, when viewed collectively, they suggest important roles for state 
office directorates and regions in terms of the delivery of system-wide 
support. 

For example, issues which require significant resource allocation for their 
successful resolution are generally beyond the capacity of individual schools 
to respond to. Major capital works, and substantial program augmentation, 
such as provision of expanded school counselling services and teaching of 
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languages other than English are examples of where a system-based 
response may be necessary. 

On the other hand, improving school-community relations is largely a 
matter for resolution at the local level. In most schools reviewed thus far, 
review recommendations tend to suggest the expansion of current good 
practices, and extending the range of areas and depth of information provided 
to parents. Likewise, extending the range of activities in which parent 
participation is sought, the greater inclusion of staff, students and parents in 
decision making, and seeking the inclusion of more representative cross- 
sections of the community are matters for the school to implement. Line 
management has a role to play in providing advice and support about 
successful strategies for enhancing communication and participation. In terms 
of fostering the successful introduction of school councils, Regions and State 
Office could play an increased role in communicating policy frameworks and 
advice, perhaps using illustrative case studies of successful practice in 
formulating the role, function and operation of school councils across the 
state. 

The implementation of student welfare programs is another area where 
practice can largely be improved by school-based initiatives. Other areas of 
student welfare, such as the provision of specialist teachers, and coping with 
extremely disruptive students are beyond the scope of individual schools 1 
response. 

In the curriculum domain, student assessment was identified as the area 
most in need of further development. Some issues in assessment, such as the 
co-ordination of practices across years and designing appropriate reporting 
formats are largely matters for school communities to resolve. However, the 
system can assist schools by articulating a consistent statewide policy on 
assessment, clarifying and promoting outcomes-based teaching, profiles and 
other implications arising from national trends in curriculum development. 
State offices also have an important role in continuing to provide information 
on how Basic Skills Test results, for example, may be interpreted and used 
appropriately for diagnostic purposes. The Training and Development 
Directorate, Regions, and Education Resource Centres have a role to play in 
providing professional development aimed at improving practice in areas 
such as the technical aspects of appropriate identification and measurement 
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of student outcomes, and encouraging the use of student outcome data in 
school planning and resource allocation. 

Many secondary schools have made impressive progress in examining 
their total curriculum provision and providing for the diverse needs of rapidly 
increasing and changing student populations. They should be encouraged to 
continue to do so. The wider education system including the Board of Studies, 
has also contributed to this re-alignment of senior secondary schooling 
through the introduction of such initiatives as Pathways and greater 
opportunities for joint school-TAFE courses. However, all of these are only 
partial solutions to a much deeper issue requiring complete re-thinking of the 
fundamental nature, purpose, structure and organisation of post-compulsory 
schooling for the next century. In this process, the Department of School 
Education has a vital leadership role to play in promoting informed debate 
and discussion. 

In the planning and decision making area, systemic support could be 
provided to schools to help them address the strategic nature of plans, advice 
on the format and content of plans, and by ensuring that schools continue to 
monitor, evaluate and adapt their plans over time. Regions and clusters could 
benefit from updated advice from central authorities on planning concepts, 
definitions of terms, and clear and unambiguous policy advice on systemic 
expectations for planning, as well as practical advice in the development of 
planning and reporting processes. 

The Quality Assurance Directorate and quality assurance processes in 
general, quite properly, sit outside of the line management structure of the 
Department. Review recommendations, and the suggestions made in this 
report, aim to address areas for strategic development of schools and the 
system, couched in terms of desired outcomes. They do not seek to prescribe 
solutions or to tell schools what-they should do or how they should do it, rather 
this advice seeks to establish what should be achieved, based on the 
collective professional knowledge of review teams. In both the school reviews 
and this synthesis of issues across the state, the identification of these trends 
and issues provides opportunities and challenges for further development to 
increase student learning outcomes. 
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Outcomes of the reporting process 

It is too early yet to make a definitive evaluation of the outcomes of the first 
systemic report, but there is some evidence of its potential for generating 
action. As part of the dissemination strategy for releasing the document, the 
education writers from both Sydney metropolitan daily newspapers were 
provided with a briefing by the Assistant Director-General (Quality 
Assurance). Both newspapers carried substantial articles about the report the 
following day which, in general, were fair representations of its content, 
although the headlines of one (Report identifies failings in schools) was not 
exactly what was intended. This was followed shortly afterwards by a lengthy 
editorial in the Sydney Morning Herald titled "How to get quality schools". This 
editorial also recognised the value of the report and adopted a positive tone in 
commenting further on one of the themes raised in the report, namely, the 

1 . . . need to keep parents fully informed with the functioning of an effe* tive, 
efficient and community-conscious school system". . . and . . .[the ne^d for] 
schools of today to 'ie more conscious than they have been of the needs of their 
consumers — the students and their parents". 

The Minister for Educe .un also released a press briefing summarising the 
report, which concluded with her interpretation that: 

"It is clear from this first review report that schools are meeting the needs of their 
local community. While there is room for improvement in some schoois, the public 
of NSW can be assured that students are receiving a world class education." 

If it has not yet influenced high level policy formation, the Review Report 
has at least entered into Ministerial consciousness and has been accepted as 
credible source of advice. 

The Dooartment of School Education also publicly responded to the 
release of the report with an article in its internal weekly newspaper School 
Education News, and as the subject of the weekly column by, the (acting) 
Director-General. These also were very positive in tone and conclusion. 

'The Department of School Education welcomed the first system wide quality 
assurance report", the Director-General Ken Boston said. 

The School Education News article summarised very briefly the areas 
covered by the report and noted system wide strengths. It also recognises that 
"There is also valuable comment on the content and organisation of the 
secondary curriculum end appropriate approaches to schoo! planing" and 
goes on to say that while the responsibility for responding to these issues will 
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generally lie at the school level, regions and state office will meet the 
challenge of providing further direction for training and development in the 
area of outcomes based teaching, and outcomes based assessment and 
reporting in general. 

The article concludes with commentary on how the Department has already 
begun to respond to the challenge of providing improved support to schools in 
the delivery of student outcomes by sharpening the focus on teaching and 
learning through the restructuring of the Department's senior management 
and the appointment of a Deputy-Director Genera! (Teaching and Learning) 
and plans for detailed program evaluations in several areas identified in the 
report as in need of further development. While these processes may have 
already been in train before the formal release of the report, they are 
supported by its findings and conclusions. It is also probable that early 
informal dissemination of the findings from the school reviews was a source of 
information in the decision to move in this direction. 

At an operational level, a summary of the findings of the report and an 
indication of their likely relevance to state office directorates was tabled at a 
meeting of the Department's state office directors. While no formal record of 
the outcome of that meeting is available, anecdotaliy, some participants 
reported a high level of interest and an unusual degree of animation in the 
ensuing discussion. At least one Directorate has sought more detailed advice 
about the implications of the quality assurance reviews for their strategic 
directions for 1994. Other presentations of the review report at various 
meetings and seminars have also created a new awareness of the power of 
the quality assurance reviews to provide public accountability and inform the 
directions taken by the system as a whole. 

CONCLUSION 

Proponents of any initiative inevitably view it as the last best hope for 
change. Quality Assurance as a management initiative is still in its infancy, 
and while the early signs of its success are encouraging, only time will tell if it 
is to survive into the next century. 

We believe that the present process will prove to be of enduring worth, 
because it attends simultaneously to the varying needs of practitioners in 
schools as well as planners and decison makers at the system level. The 
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inspectorate, which served the system well for so many years, finally became 
redundent because the approach to accountability derived from the 
inspector's position of authority was never really compatible with their 
developmental role, nor did the structure match the needs of a self-managing, 
devolved school system. In terms of their contribution to systemic 
improvement, the time lag between performance of inspectors 1 work in 
schools and the aggregation of information gleaned as a part of this process 
through the formal and informal channels of communication available to them 
made it difficult for others to perceive what value they were adding to the 
system. The plan for systemic reporting outlined above, and the school review 
model from which it draws its data offers a degree of flexibility which will allow 
it to continue to adapt to the changing circumstances schools will face in the 
decade ahead. 

At either school or system level, the provision of information will continue to 
have currency as the process of reform and devolution continues because a 
constant flow of information is necessary to move past rule governed 
management and regulated teacher behaviour to a position where 
professional judgement or good management is the primary means by which 
courses of action are selected. While it may be an article of faith that people 
will act reasonably if they have enough information on which to base their 
decisions, th<3 process of systemic reporting described above, and plans for 
effective dissemination of information established by the Quality Assurance 
Directorate will at least ensure that information about the status and outcomes 
achieved by schools is brought into the policy-making arena. 
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